
AIAA-2003-0409 

Uncertainty in Computational Aerodynamics 

J. M. Luckring, M. J. Hemsch, J. H. Morrison 
NASA Langley Research Center 
Hampton, Virginia 


41st AIAA Aerospace Sciences Meeting & Exhibit 

6-9 January 2003 
Reno, Nevada 


For permission to copy or republish, contact the American Institute of Aeronautics and Astronautics 
1801 Alexander Bell Drive, Suite 500, Reston, VA 20191-4344 



AIAA-2003-0409 


Uncertainty in Computational Aerodynamics 


J. M. Luckring , M. J. Hemsch , J. H. Morrison 
Aerodynamics, Aerothermodynamics, and Acoustics Competency 
NASA Langley Research Center 
Hampton, Virginia 


ABSTRACT 

An approach is presented to treat computational 
aerodynamics as a process, subject to the fundamental 
quality assurance principles of process control and 
process improvement. We consider several aspects 
affecting uncertainty for the computational 
aerodynamic process and present a set of stages to 
determine the level of management required to meet 
risk assumptions desired by the customer of the 
predictions. 



NOMENCLATURE 

CA 

Computational Aerodynamics 

C| 1E iax 

maximum section lift coefficient 

r/c 

leading-edge-radius to chord ratio 

Re 

chord Reynolds number, U QO c/v 

SPC 

Statistical Process Control 

V&V 

Verification and Validation 


INTRODUCTION 


Computational aerodynamics has a rich and long 
history as methods have evolved hand-in-glove with 
the evolution of high-speed computing. These 
methods are anchored in fluid mechanical and 
aerodynamic theory, and a hierarchy of techniques 
has been developed over many decades that differ 
primarily in fidelity, flow physics representation, and 
computational efficiency. Although most current 
work is focused on Computational Fluid Dynamics 
(CFD), Computational Aerodynamics (CA) embraces 
a broader scope than CFD. Computational 
Aerodynamics has as an end objective the 
development of full-scale vehicle capability as well as 
the a-priori prediction of the full-scale vehicle 
properties. As such, this entails the use of one or 
more CFD methods, reduced physics methods, 
prediction or extrapolation techniques, and 
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calibration information that can come from 
experimentation. The particular suite of methods 
used for CA is driven by a combination of technical 
issues (such as method capability) and business 
factors (such as schedule, resource limitations, 
historical approach, etc.) 

Two contrasting views toward computational 
aerodynamics are presented in Figure 1 [Zang 2002, 
and Wahls 2002] in the context of computational 
fluid dynamics. In this figure, CFD utility is 
illustrated for a notional full-envelope range of 
vehicle operating conditions. Figure la illustrates the 
current or traditional approach to CFD that is 
characteristic of deterministic CFD. Here, CFD is 
illustrated to provide trusted predictions (acceptable 
accuracy) for both low-speed and cruise attached- 
flow aerodynamics. This has resulted from sustained 
modeling efforts of these flows. However, separated 
and unsteady flow effects can be important for a 
significant fraction of the overall flight envelope. 
Modeling for separated and unsteady flows has 
proven to be a daunting task (as have experimental 
studies), and as such CFD has not demonstrated 
sufficient accuracy for these flows. 

An alternate view is presented in Figure lb and is 
referred to as uncertainty-based CFD. This view is 
the focus of the present paper. Emphasis with this 
approach is placed upon quantifying the uncertainty 
of computational aerodynamic methods throughout 
the flight domain. This means quantifying not only 
the value of some prediction (e.g., lift), but also the 
error bounds and confidence level associated with 
such a prediction, and doing so in the context of a 
well-defined and repeatable process. Because 
accuracy requirements are not necessarily as stringent 
in the whole envelope as they are at cruise conditions, 
the uncertainty-based CFD results could provide the 
confidence needed to address off-cruise-condition 
requirements. Such information can also then be 
related to risk issues for a decision maker in the 
context of vehicle design, development, or 
modification processes. 
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The overall goal for computational aerodynamics 
uncertainty therefore is to establish a process to 
produce credible, quantified, unambiguous, and 
enduring statements of computational uncertainty for 
aerodynamic methods. We also anticipate that, with 
such a process, the quality of aerodynamic 
predictions could be assured the first time the process 
is performed, such that the current practice of 
adjusting computations to match experimental 
findings could be greatly reduced or even eliminated. 
Under these circumstances, computational 
aerodynamics could be more heavily relied upon for 
aerodynamic predictions. 

To achieve confidence in aerodynamic predictions 
will require confidence in error bounds. We need to 
quantify the error bounds as they presently exist and 
also understand the sources and mechanisms of the 
error. The impact of these uncertainties is a separate 
consideration and more the concern of the customer 
of the computational predictions. Mission priorities 
then dictate which error bounds warrant further 
research and reduction. 

Here we are drawing a clear distinction between the 
Voice of Customer and the Voice of the Process as 
enunciated by Wheeler and Chambers [1992], The 
Voice of the Customer is the specifications required, 
including tolerances. The Voice of the Process for an 
overall computational aerodynamics result would be 
what the computational process produces, including 
uncertainties, as long as the process continues to 
operate stably. These two Voices must be considered 
and dealt with separately if the notion of uncertainty- 
based CA is to be useful. It should be noted that the 
Voice of the Process has no meaning if the process is 
not stable [Wheeler and Chambers 1992]. 

For example, the AIAA Applied Aero Technical 
Committee conducted a Drag Prediction Workshop in 
2001 to evaluate CFD prediction of transonic cruise 
drag predictions for transport configurations. 
Workshop participants calculated forces and 
moments on the DLR-F4 transport model at the 
nominal cruise condition. This configuration was 
chosen because there were data available from three 
different wind tunnels. A total of 35 flow solutions 
were contributed from 14 different CFD codes 
ranging from research codes to commercially 
available codes. Hemsch [2002] performed a 
statistical analysis of the results and determined that 
the CFD results exhibit a 2-sigma confidence interval 
of roughly +/- 40 counts of drag (1 count of drag 
corresponds to a C D change of 0.0001) at the cruise 
condition and the experimental data exhibits a 2- 


sigma confidence interval of roughly +/- 8 counts. 
These results represent the Voice of the Process. 
Aircraft manufacturers report that they require drag 
predictions within +/- 1 count of drag. This represents 
the Voice of the Customer. Clearly, neither the CFD 
predictions nor the wind tunnel data are capable of 
meeting this requirement. 

In almost all cases, process improvement requires two 
steps: first, elimination of assignable causes in order 
to create a stable, predictable process and second, 
modification of the process, if necessary, to reduce 
the residual variation. The DPW workshop 
participants identified that the grids used for the 
predictions were inadequate. For this particular 
application, lack of grid convergence led to an 
unpredictable process (14% of the solutions were 
outside the statistical limit boundaries of the core 
solutions) and to process variation considerably 
larger than required (+/- 40 counts versus +/- 1 count 
respectively.) This issue is being addressed in the 
second AIAA Drag Prediction Workshop scheduled 
for the summer of 2003. The results of the first 
workshop clearly indicate the need for more rigorous 
processes to quantify and manage uncertainty for 
computational aerodynamics. 

Computational uncertainty assessments embrace a 
variety of technologies and issues. In the next 
section, we review some critical CA uncertainty 
process considerations. In the following section, we 
discuss fundamental management issues for 
implementing the CA uncertainty process in different 
risk assumption environments. 

PROCESS CONSIDERATIONS 

Some considerations of prediction will first be 
reviewed. This is followed by a discussion of 
inferences that may be drawn from the computational 
simulation. Finally, a list of verification, validation 
and uncertainty elements is presented. 

Prediction to Unvalidated Conditions 

Prediction has at least three different usages in terms 
of computational aerodynamics. First, there is a 
computational prediction for a point-wise match to 
data from an independent source. Here, we often see 
an a-posteriori comparison of a computation (usually 
CFD) against experiment. Second, prediction is used 
in the sense of interpolation to conditions that fall 
within those already anchored by a comparison 
between the computation and benchmark information. 
Third, prediction is used in the sense of an 
extrapolation to conditions that fall outside the 
domain for which the computation is anchored by 
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independent data. This last form of prediction is the 
most desired and, unfortunately, usually comes with 
the highest uncertainty. 

This extrapolation form of prediction (e.g., from 
ground-based to full-scale Reynolds number) entails 
many other considerations separate from traditional 
verification and validation issues. As such there can 
be separate and distinct sources of uncertainty 
associated with the prediction process itself. These 
include issues that could range from ground-based 
test technique concerns (e.g., wind-tunnel wall 
interference, etc.) to full-scale vehicle medium 
uncertainty, such as for planetary entry. 

Trucano [1998] brought a noteworthy perspective 
toward prediction by emphasizing the consequences 
associated with prediction uncertainty. This helps 
establish how much confidence is needed in the error 
estimate. Consequences in the prediction (or full- 
scale) domain need to be considered heavily in setting 
priorities for which problems get worked, and how 
they are worked. There may be high consequence 
issues, with relatively simple physics, that would be 
more important to address than lower consequence 
issues that would none-the-less require sophisticated 
computational technology. For example, late in the 
Shuttle development program there were lingering 
concerns for tile loads, including those due to the 
flow in the gaps between the tiles. Dwoyer et al 
[1982] demonstrated that this flow could be modeled 
with the Stokes flow approximation and obtained 
sufficiently accurate simulations in time to contribute 
to the resolution of this program concern. 

The extrapolation form of prediction from 
computational aerodynamic methods naturally leads 
to inference space considerations. The nature of the 
inference space, as a context to the computational 
predictions, can significantly affect the underlying 
uncertainty of these predictions. 

Inference Space 

Aerodynamic flow domains provide a useful context 
to perform aerodynamic inferences. This leads to the 
more general concept of a physics-based inference 
space. 

A simple example of flow domain considerations is 
taken from Polhamus [1996] for airfoil stall 
characteristics. Figure 2. In this work Polhamus 
retained the stall characterization developed by Gault 
[1957]. Figure 2a shows three of these distinctions as 
being: i) thin-airfoil stall, ii) leading-edge stall, iii) 
trailing-edge/leading-edge stall, and not shown is iv) 


trailing-edge stall. As the flow sketches indicate, the 
flow physics of each of these classifications is quite 
different, and the consequences on the lift properties 
near C] >max are also quite different. The ability of a 
computational method to predict one of these classes 
of stall would not necessarily imply that it could 
predict the other classes since the underlying flow 
physics are different in each case. 

Polhamus modeled the airfoil stall flow types in terms 
of two parameters, airfoil leading-edge radius and 
chord Reynolds number as shown in Figure 2b. The 
data for this figure are from the NACA 6-series of 
airfoils, and, subject to the available data, Polhamus 
estimated boundaries between the various stall 
classifications. It is hypothesized that one can have 
increased confidence extrapolating a validated (or 
calibrated) computational method in the parameter- 
space variables (r/c and R c in this example) to 
conditions beyond the validation conditions so long 
as a domain boundary is not crossed. The domains, 
as shown in Figure 2b, are examples of what 
constitutes a physics-based inference space. It is 
interesting that with this view toward a physics-based 
inference space, a direct link between basic flow 
physics (e.g., turbulent reseparation) and aggregate 
aerodynamic properties, like Q max , are readily 
established. Having the critical physics linked to 
prediction metrics (like C lmax ) is not only crucial to 
method validation, but also should greatly reduce the 
uncertainty in these predictions. 

The fact that there are flow domains is nothing new. 
Examples can be found for many flow considerations, 
such as the work of Stanbrook and Squire [1965] and 
later Miller and Wood [1985] who established 
domains of supersonic leading-edge vortex separation 
and subsequent classes of shock-vortex structures. 
What is new is the approach of exploiting these 
domains, and building new domain knowledge, in the 
context of computational uncertainty and inference of 
prediction uncertainty. These domains, or physics- 
based inference spaces, generally exist in a similarity 
space within which we hypothesize that extrapolation 
becomes tractable with greatly reduced uncertainty as 
compared to other means. 

Transformation from primitive to similarity variables 
generally results in reduced dimensionality for both 
independent and dependent variables. Although a 
simple example of this reduced dimensionality is 
given by transonic similarity, examples for more 
complex flows have also been developed. For 
example, Stahara [1981] used the concept of strained 
coordinates [Lighthill 1949] to demonstrate the utility 
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of similarity-based methodology for predicting 
transonic airfoil aerodynamics with or without flow 
discontinuities. However, the flows must be 
topologically similar. 

The method of strained coordinate used by Stahara 
[1981] can be useful in determining the uncertainty of 
a solution where no validation data exists, provided 
that the flows are topologically similar and the 
interpolation remains in a physics-based inference 
space. First, determine the uncertainty at two points 
within the physics based inference space where 
validation data exists using error propagation or In- 
version testing. Then apply the method of strained 
coordinates to interpolate this uncertainty to the new 
solution point. 

Crucial knowledge for a physics-based inference 
space to be useful comes down to i) knowledge of the 
underlying flow physics to the aerodynamic topic of 
interest (which could be anchored in theoretical or 
experimental considerations) and ii) knowledge of the 
boundary region for said physics. This emphasis on 
physics-based boundary regions between 
aerodynamic flow states could present a new view 
toward aerodynamic testing for the purposes of code 
calibration and validation. A simple example is the 
establishment of the boundary between attached-flow 
cruise aerodynamics and separated flow for a given 
vehicle class. Data for the onset of separation would 
be useful for enhancing prediction confidence within 
the attached flow domain. The data could also guide 
the development of boundary prediction methodology 
to further reduce the uncertainty associated with state- 
change flow physics. 

If a physics-based inference space is bounded, that is, 
if the boundary is established (say, experimentally) 
then it may be possible to validate the CA code within 
that domain. However, in an unbounded inference 
space a CA code can at best be validated at discrete 
flow conditions. Predictive capability within the 
bounded physics-based inference space should be 
possible with much higher confidence. These ideas 
are abstracted in Figure 3. Computational uncertainty 
near domain boundaries will certainly be higher than 
in the interior, not only because of the more complex 
state-change physics, but also because of the inherent 
uncertainty of the conditions causing this change of 
state (i.e., the parameter space range of the 
boundary). Physics to model and eventually predict 
these boundaries will in most cases be the hardest 
capability to be developed and thus may depend the 
most heavily on experimental technology or more 
advanced numerical simulation. 


The boundary may be analogous to the intermediate 
solution in an asymptotic expansion. It provides a 
neighborhood through which the solutions match 
(consider, for example, the overlap region in the log- 
law/law of the wall from turbulent boundary layer 
theory). The scale/extent of the boundary will be 
different for different flows, and this scale itself will 
be a metric of interest. The boundaries of the 
inference spaces more than likely manifest higher- 
order flow physics with corresponding implications 
on computational uncertainty - the closer a solution is 
to a boundary the higher the likely uncertainty, 
especially if the location of the boundary cannot be 
predicted exactly. 

The approach of validating computational 
aerodynamic techniques in the context of a physics- 
based inference space could have significant 
consequences for inferences that must be drawn in 
extrapolating from experimental test conditions to 
full-scale application conditions. Figure 4 illustrates 
that testing generally includes sub-scale conditions 
from which inferences must be drawn for the full- 
scale vehicle application. (Easterling [2001] has 
published a more sophisticated treatment of this 
topic). As one practical example, consider 
extrapolating in Reynolds number from wind tunnel 
to flight conditions. As illustrated in Figure 4, an 
extrapolation in the sense of parameter-based 
inference space can essentially become an 
interpolation in the sense of a physics-based inference 
space. 

The nature of ground-based testing would most likely 
be altered to support the physics-based inference 
space approach to extrapolation. For the Reynolds 
number scaling example, wind tunnel data would 
need to be obtained at sufficiently high values to 
assure that the full-scale flow physics were fully 
established, albeit at subscale values. Computational 
techniques, validated with these data, and in the 
context of the physics-based inference space (e.g., 
fully turbulent transonic attached flow) could then be 
used as shown in Figure 4b. It is anticipated that this 
approach could greatly reduce the uncertainty 
associated with full-scale aerodynamic predictions. 

Two facets of the physics-based inference approach 
to computational aerodynamic uncertainty could 
contribute to reduce ground-based testing 
requirements. First, the coupling of the physics-based 
inference space approach and aerodynamic similarity 
principles could be highly desirable as a means to 
reduce test conditions. Second, Modern Design of 
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Experiments principles could also be highly effective 
within the physics-based inference space to further 
reduce test condition requirements [DeLoach 1998]. 

Verification and Validation Elements 

The AIAA definitions for verification and validation 
are given as follows [AIAA Guide]: 

Verification: The process of determining that a 
model implementation accurately represents the 
developer’ s conceptual model and the solution to the 
model. 

Validation: The process of determining the degree to 
which a model is an accurate representation of the 
real world from the perspective of the intended uses 
of the model. 

The quantification of uncertainty in a numerical 
prediction can be broken down into several sources. 
We use the following four sources: method V&V, 
process control, parameter uncertainty, and model 
form uncertainty (Figure 5). Here we use method 
V&V to distinguish sub-process V&V (such as for a 
CFD code) from the overall V&V process as defined 
in the AIAA guide. Method V&V and process 
control are primarily concerned with the 
quantification and control of numerical errors and 
uncertainty. Parameter and model form uncertainty 
are concerned with quantification of uncertainty in 
the physical process being modeled. 

Method V&V. Code verification and validation are 
necessary first steps in quantifying uncertainty. 
Roache [1997] distinguishes between verification and 
validation of a code and verification of a solution. 
Roache [1998] states “Verification is completed (at 
least in principle, first for the code, then for a 
particular calculation) whereas Validation is ongoing 
(as experiments are improved and/or parameter 
ranges are extended).” 

Code verification is a process to ensure that the 
model equations are solved correctly. The Method of 
Manufactured Solutions [Roache 1998] is a powerful 
technique for code verification. A non-trivial 
solution is specified and forcing functions added to 
the governing PDEs to satisfy this solution. A grid 
convergence study then verifies the order of accuracy 
of the code [Salari and Knupp 2000], 

However, the use of a verified code is insufficient. A 
grid convergence study is required to estimate the 
error on new calculations. Additionally, a grid 
convergence study should be used to check that the 
order of convergence on the new prediction matches 


the advertised accuracy of the code. A reduced order 
of accuracy is an indication of an error in the 
calculation. Roache [1997] gives several examples of 
calculations where an esoteric error in a new code 
application resulted in a reduced order of accuracy. 

Grid convergence studies (extending into the 
asymptotic range) using verified codes are the most 
common and direct method to quantify numerical 
uncertainty. 

Process Uncertainty. CFD code application remains 
a labor-intensive activity. The process requires 
specification of (usually) complex geometry and 
discretization of the flow volume. The practitioner 
must make appropriate decisions about the flow 
physics (incompressible or compressible, laminar, 
transitional or turbulent, etc.) and choose appropriate 
physical models. Budgets and deadlines limit the 
resources available for the analysis and prevent the 
analyst from exploring alternate models. Simple 
errors in input decks can invalidate the results. 

Process control, in the sense of modern quality 
assurance, not only minimizes the blunders due to 
misapplication of codes and simple user errors, but it 
also enables the creation and management of a stable, 
predictable process with known variation - the target 
of Figure lb. We believe that generally available best 
practices (e.g., ERCOFTAC [2000], Chen [2002]) 
will provide a foundation for developing and 
establishing such processes. 

Parameter Uncertainty. The conceptual model that 
is implemented in a computer code is an idealization 
of the physical world that it is used to model. For 
example, manufactured vehicles vary from the design 
specifications with each vehicle being slightly 
different, aircraft deformations under load are 
difficult to accurately measure or predict, physical 
properties vary or are difficult to estimate. The 
aeroelastic deformation of a wing in flight can have a 
large effect on the resulting flowfield. It is important 
that the actual aeroelastic shape of the aircraft be 
incorporated in the numerical model to accurately 
predict the flow. The uncertainty in this deformation 
can be accommodated using error propagation 
techniques. Additionally, there is uncertainty in the 
value of physical constants (e.g. thermal conductivity, 
viscosity, specific heats, etc.) that can be important 
for some classes of problems. Several techniques 
exist to propagate the effects of these variations on 
system response; sensitivity analysis, Monte-Carlo, 
and polynomial chaos among them [Hills and 
Trucano 1999, Walters 2003]. For example, Pelletier 
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et al. [2002] showed the effect of geometric 
uncertainty and physical constant uncertainty on the 
validation study of a laminar free -convection problem 
with variable fluid properties. 

Model Form Uncertainty. Validation is 
confirmatory evidence that the conceptual model is an 
adequate representation of the physical process for 
the intended use of the model. Model form 
uncertainty is a quantification of a model’s predictive 
accuracy. 

Current modeling efforts focus on unit problems. 
Models are typically developed from theory, 
validated on unit problems and then tested on more 
complex configurations. The resulting model 
becomes widely accepted when a sufficiently large 
number of unit problems and configurations have 
been computed with results equivalent or better than 
the previous best model. This ad hoc procedure for 
model acceptance does not provide an estimate of 
uncertainty for model predictions. A framework 
needs to be developed to provide quantification for 
model form uncertainty. Very little work has been 
done in this area [Hills and Trucano 1999, Luis and 
McLaughlin 1992], 

There is some ambiguity whether certain sources of 
uncertainty should be classified as parameter 
uncertainty or model form uncertainty. We have 
chosen to classify physical constants under parameter 
uncertainty. However, we feel that model 
coefficients, such as used in turbulence models, are 
part of the model and must be considered as part of 
model form uncertainty. 

Interrelationships among Uncertainty Sources. 

Although the uncertainty sources of Figure 5 each 
have unique aspects, it is important to acknowledge 
that they also interact and affect one another. Our 
current view of these interactions is shown in Figure 
6. Also included in this figure are a number of 
additional key topics (not shown in Figure 5) that 
contribute to aerodynamic prediction uncertainty. 

PROCESS MANAGEMENT 

The two sections below provide an approach for 
moving toward uncertainty-based computational 
aerodynamics. The first section regards relationships 
between computational output and Statistical Process 
Control. The second section addresses management 
and risk considerations. 


The Three Aspects of any CA Output and Their 
Relationship to Statistical Process Control 

In the CA uncertainty process, there are 
fundamentally two aspects to every number of 
importance that is generated as part of the code 
output: (1) the generated value itself and (2) the 

difference between the so-called true value and the 
generated value, i.e. the error. However, in all 
practical cases of interest, the true value is not known 
and, hence, the error is also unknown. To deal with 
this issue statistically, we would define a 
computational process. But now there are necessarily 
three aspects to every process output value: (1) the 
average of the sample realizations of the process, 
which is the estimate of the population mean, (2) the 
sample standard deviation, which is the estimate of 
the population standard deviation, and (3) the number 
of realizations, which reflects how well the 
population parameters are known. 

The CA uncertainty process is concerned with aspects 
(2) and (3) whether they are known qualitatively or 
quantitatively. The risk for a decision maker is not 
associated with the value itself nor even the size of 
the standard deviation since modern probabilistic 
methods, at least in principle, can propagate those 
effects, but rather how well the second value is 
known. 

For the last four decades, methods for building 
confidence in the second and third aspects have been 
developed for precision metrology by the National 
Institute for Standards and Technology (NIST) and its 
predecessor, the National Bureau of Standards (NBS) 
[Eisenhart 1969], Those methods, which are based 
on the notions of statistical process control (SPC) 
used in manufacturing [Wheeler and Chambers 
1992], have been recently adapted to force and 
moment testing in wind tunnels [Hemsch et al. 2000]. 
We suggest that the CFD community consider such 
methods for improving the CA uncertainty process. 
A qualitative definition of SPC for measurement 
processes is given below followed by a more 
mathematical version [Belanger 1984]: 

A measurement process is in a state of statistical 
control if the amount of scatter in the data from 
repeated measurements of the same item over a 
period of time does not change with time and if there 
are no sudden shifts or drift in the data. (Qualitative) 

A measurement process is in a state of statistical 
control if the resulting observations from the process, 
when collected under any fixed experimental 
conditions within the scope of the a priori well- 
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defined conditions of the measurement process, 
behave like random drawings from some fixed 
distribution with fixed location (mean) and fixed 
scale (standard deviation ) parameters. (Quantitative) 

For the purposes of CA uncertainty, we would 
consider the sources of error to be different grids, 
observers, codes, turbulence models, etc. 

Management of the CA Uncertainty Process for 
Decision Makers and for Resource Allocation 

It has been our experience that effective response of 
stakeholders (funders, managers and workers) to CA 
uncertainty issues is largely a function of their 
understanding of the present state of their CA 
uncertainty process either for an individual software 
development project or for a vehicle design effort or 
for the organization as a whole. It may be less well 
known that it is also a function of their knowledge of 
potential CA uncertainty stages. We propose in this 
section a particularly simple way to consider and 
display those stages to stakeholders. Knowing where 
the organization or project is relative to those stages 
and the actions required for each one makes it 
possible to allocate resources effectively, particularly 
if the risk state that the customer is willing to assume 
is clear. 

It is convenient to think of the decision maker as the 
customer of the CA uncertainty process and their 
desired risk assumption level as the Voice of the 
Customer, a term from the Statistical Quality Control 
literature [Wheeler and Chambers 1992]. In 
manufacturing, the Voice of the Customer might be a 
specified tolerance interval defined by a design 
department. Similarly, we could call the actual 
performance of the CA uncertainty effort, the Voice 
of the Process. Again, in manufacturing, the Voice of 
the Process might represent the actual variation in the 
output from a series of manufacturing steps [Wheeler 
and Chambers 1992]. 

Below we offer five representative CA uncertainty 
stages and correlate them with decision-maker risk 
assumption level. Readers who are familiar with the 
Software Capability Maturity Model [Paulk et al 
1993] or the Process Maturity Model [Gardner 2001] 
will recognize that the CA uncertainty stages share 
some, but not all, of their features. Consider the five 
possible stages shown in Figure 7. They are: 

1. No systematic uncertainty effort. Activity in 
this stage is best described as ad hoc. Success 
depends upon the experience of particular 
individuals acting alone or as a team. It would 


be reasonable to expect to find research efforts at 
this level, particularly when entirely new code 
functionality is being attempted. For this level, it 
is impossible for decision makers to assess their 
risk assumption level. 

2. Uncertainty management. At this stage, 
considerable effort is made to keep the error of 
code predictions within target bounds but any 
uncertainty statements are best characterized as 
qualitative. For example, if an uncertainty band 
is given, the precise coverage level and how well 
it is known are left unstated. It is probable that, 
when resources are not available to drive grid 
convergence into the asymptotic region, this is 
the best level attainable unless the processes have 
been demonstrated to be in statistical control 
[Wheeler and Chambers 1992], The Boeing 
Company has developed a particular philosophy 
for this stage and shown its value for the airframe 
aero design process [Rubbert 1998], At this 
level, a decision maker would typically have 
precedent as a guide in assessing the risk 
assumption level. It is also likely that the 
decision maker will include safety factors and/or 
margins to reduce the consequences of the 
insufficiently known uncertainty. For this stage 
there is basic project management to repeat 
earlier successes on projects with similar 
applications. 

3. Reported uncertainty. At this stage, the CA 

uncertainty process is capable of reporting a 
quantitative uncertainty band, together with the 
coverage level and how well it is known, for the 
domain of the validation effort, past and present. 
It is unlikely that this level can be achieved 
without grid convergence to the asymptotic 
region and without experimental efforts that are 
specifically designed for code validation 

[Oberkampf and Trucano 2002]. Achieving this 
level would seem to require process 

documentation, standardization and integration 
across the organization with all projects using 
approved, tailored versions of the organization’s 
standard process for maintaining the CA 
uncertainty process. Achieving this stage would 
allow decision makers to reduce the safety 
factors and/or margins required at stage 2, 
particularly for predictions which are somewhat 
close to the previously explored inference space. 

4. Reported prediction error. This stage is 
distinguished from stage 3 by the ability to 
estimate the uncertainty band associated with a 


American Institute of Aeronautics and Astronautics 



AIAA-2003-0409 


prediction outside the validation domain, 
together with a statement of coverage level and 
how well it is known. Clearly, this would 
require, in addition to the requirements of stage 
3, thorough understanding of model form error in 
the validation region as well as in the prediction 
inference space [Easterling 2001], (See the 
Inference Space section above for possible ways 
to carry out such studies.) For this stage, it may 
be possible for decision makers to use the 
uncertainty bands to assess risk without safety 
factors or margins. However, so little work has 
been done in this area that it is not clear if such a 
change will ever be warranted. But, for this 
stage, it seems appropriate to consider that the 
organization that is producing the predictions is 
assuming most of the risk. 

5. Certification and Protocols. At this stage, the 
decision maker should be assuming none of the 
risk because the producing organization is 
guaranteeing that the prediction uncertainty band 
and its coverage level are known. See Lee 
[1998] and Pilch [2000] for descriptions of such 
activities. 

It is important to realize that moving from a lower to 
higher CA uncertainty stage requires considerable 
effort on the part of all stakeholders. It actually 
requires a paradigm shift that is impossible to 
appreciate and understand until the stage has been 
achieved. Training and education of all of the 
stakeholders is essential to make such a shift possible. 
Achieving the higher stages requires attempting to 
work at those levels and providing the necessary will 
and resources. 

CLOSING REMARKS 

Computational aerodynamics is a rapidly evolving 
field that has been widely accepted in engineering 
analysis and design. However, uncertainty estimates 
are required for computational aerodynamics users 
and decision makers to analyze and manage risk. In 
this paper we have enumerated the stages of CA 
maturity. Each organization will be required to 
determine their own needs and work to accomplish 
the maturity level adequate to meet their objectives. 

Various elements of CA uncertainty have been 
outlined. A method to estimate uncertainty at flow 
conditions when there is no experimental validation 
data has been outlined using the concept of physics 
based inference space and the method of strained 
coordinates. 


A tremendous amount of work remains to develop 
techniques to calculate model form uncertainty. 
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CFD Accuracy Unknown 



\ / 

CFD Accurate 


a) Deterministic CFD 



b) Uncertainty-based CFD 

Figure 1.- Alternate views toward computational aerodynamics. 
From Zang [2002]and Wahls [2002] 
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Figure 2.- Airfoil stall inference spaces. From Polhamus [1996] 
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Parameter Space 

Figure 3.- Relationships among calibration, validation and inference spaces. 




a) Parameter-based extrapolation 


b) Physics-based extrapolation 


Figure 4.- Computational prediction. 
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Model Form Uncertainty 


Parameter Uncertainty 


Have I modeled 
the right physics? 


How are my 
computed results 
affected by 
input choices? 



Goal 



Produce credible, quantified, unambiguous, 
and enduring statements of computational 
uncertainty such that the quality of aerodynamic 
predictions is assured “first time out of the box" 


✓ 


V 


Have I solved my 
equations correctly? 

Have I matched 
benchmark results? 


Customer Issues 

What are the consequences 
of this uncertainty? 
What can be done 
to reduce this uncertainty? 


Is the user trained 
and following 
stipulated procedures? 


Method V&V Process Uncertainty 

Figure 5.- Verification and Validation elements for computational aerodynamics. 



Figure 6.- Uncertainty element interactions. 
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All Risk Assumed by Decision Maker None 

Figure 7.- Mapping computational uncertainty stages onto decision-maker risk. 
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